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INTRODUCTION 

Broadly  speaking,  any  orderly  analytic  study  designed 
to  help  a  decisionmaker  identify  a  preferred  course  of 
accion  from  among  possible  alternatives  might  be  termed  a 
systems  analysis.  As  commonly  used  in  the  defense  community, 
the  phrase  "systems  analysis"  refers  to  a  formal  study 
intended  to  advise  a  decisionmaker  on  the  policy  choices 
involved  in  such  matters  as  base  operations,  weapon  develop¬ 
ment,  force  posture,  or  the  determination  of  planning 
objectives . 

The  idea  of  an  analysis  to  provide  advice  is  not  new 
and,  in  concept,  what  must  be  done  is  simple  and  rather 
obvious.  One  looks  at  the  entire  problem  in  context  and 
compares  alternative  choices  In  the  light  of  their  possible 
outcomes.  Character is tically,  such  an  analysis  requires, 
in  the  most  general  view,  three  sorts  of  inquiry,  any  of 
which  can  modify  the  others  as  the  work  proceeds.  There 
is  a  need,  first  of  all,  for  a  systematic  investigation  of 
the  decisionmaker's  objectives  and  of  the  relevant  criteria 
for  deciding  among  the  alternatives  that  promise  to  achieve 
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these  objectives.  Second.,  tbe  alternatives  need  to  be 
compared,  usually  in  terms  of  their  cost,  effectiveness, 
timing,  or  risk — using  a  quantitative  model  insofar  as  it 
can  be  made  representative  of  the  situation.  Finally, 
there  must  be  an  attempt  to  design  better  alternatives  and 
select  other  goals  if  those  examined  are  found  wanting. 

Even  though  the  concept  is  simple,  in  practice  the 
actual  conditions  of  the  analysis  pose  many  problems.  The 
fact  that  systems  analysis  still  lacks  an  adequate  theoret¬ 
ical  foundation  and  is  thus  more  an  art  than  a  science 
implies  that  it  is  pointless  to  expect  success  to  come 
merely  by  following  a  set  of  definite  rules.  The  environ¬ 
ment,  the  circumstances  under  which  the  advice  is  offered, 
may  compound  the  difficulties,  as  witness  the  decisionmaker 
who  is  in  such  haste  for  an  answer  that  he  actually  makes 
it  impossible  for  the  analyst  to  conduct — or  complete — the 
study  properly.  The  analyst  himself,  after  a  period  of 
fruitful  work  may  have  become  so  entrenched  in  his  organiza¬ 
tion  and  so  saturated  with  a  decisionmaker's  ideas  that  he 
has  lost  his  independent  viewpoint.  The  questions  investi¬ 
gated  almost  always  involve  more  than  the  efficient  allocation 
of  resources  among  known  and  accepted  uses;  often  the  objec¬ 
tive  or  goals  cf  the  action  to  be  taken  must  be  determined 
first,  and  it  may  be  more  difficult  to  decide  what  ought  to 
be  done  than  how  to  do  it.  Indeed,  the  very  aim  of  analysis 
itself,  which  is  to  suggest  a  course  of  action  to  someone 
else,  tends  to  Introduce  all  the  difficulties  and  contra¬ 
dictions  associated  with  »v\ue  concepts,  human  behavior, 
and  the  communication  of  ideas. 

I  see  these  problems  as  two  types:  those  inherent 
in  all  analysis  of  choice  and  those  that  are  to  some  extent 
of  our  own  making.  Let  me  begin  with  a  discussion  of  three 
problems  of  the  first  type:  the  selection  of  measures  of 
effectiveness;  the  treatment  of  uncertainty;  aid  the  hand¬ 
ling  of  considerations  that  cannot  be  readily  quantified. 
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Later ,  I'll  talk  about  problems  that  arise  because  we 
introduce  bias,  give  insufficient  attention  to  problem 
formulation,  and  sometimes  forget  the  proper  role  of 
analysis . 

MEASURES  OF  EFFECTIVENESS 

In  a  1966  report  designed  to  accompany  the  bill  author¬ 
izing  funds  for  the  Defense  Department,  the  Rouse  Armed 
Services  Committee  remarked  that  "...the  almost  obsessional 
dedication  to  cost  effectiveness  raises  the  spector  of  a 

decision  maker  who  ...  know  the  price  of  everything  and 

# 

the  value  of  nothing.  This  remark  touches  on  the  most 
serious  problem  in  the  application  of  systems  analysis  to 
national  security  problems — the  determination  of  valid 
measures  of  effectiveness  and  the  associated  one  of  sensible 
criteria  for  choice. 

Not  only  are  national  military  objectives  likely  to  be 
multiple,  ill-defined,  and  conflicting,  but  the  measures  of 
their  attainment  are  likely  to  be  inadequate  approximations 
at  best.  For  indicating  the  attainment  of  such  vaguely 
defined  objectives  as  deterrence  or  victory,  it  is  even 
hard  to  find  measures  that  point  in  the  right  direction'. 
Consider  deterrence,  for  instance.  It  exists  only  in  the 
mind — and  in  the  enemy's  mind  at  that.  We  cannot,  therefore, 
use  some  "scale  of  deterrence"  to  measure  the  effectiveness 
of  alternatives  we  hope  will  lead  to  deterrence,  for  there 
is  no  such  scale.  Instead,  we  must  use  such  approximations 
as  the  potential  mortalities  that  might  be  suffered  if  war 
were  to  come.  Consequently,  even  if  a  comparison  of  two 
systems  indicated  that  one  could  inflict  50  percent  more 
casualties  on  the  enemy  than  the  other,  we  still  could  not 
conclude  that  this  meant  the  system  supplies  50  percent 
more  deterrence.  In  fact,  since  in  some  circumstances  it 
may  be  important  not  to  look  too  threatening,  we  might  even 

#House  Rect.  1536  AuthorlgingDe fenae  Procurement  and 
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argue  that  the  system  capable  of  inflicting  the  greatest 
number  of  casualties  may  provide  the  least  deterrence! 

The  inability  to  determine  good  measures  of  effective¬ 
ness  is  a  severe  limitation  on  the  usefulness  of  analysis. 
Suppose  we  are  seeking  to  determine  the  characteristics  of 
future  tactical  aircraft.  If  a  single  measure  such  as 
weight  of  ordinance  delivered  is  used,  the  aircraft  are 
likely  to  look  like  the  B— 52  or  the  C5.  Some  combination 
is  required  that  cakes  account  of  all  the  characteristics 
that  are  important  to  the  effectiveness  of  such  aircraft 
— qualities  such  as  speed,  bombing  accuracy,  runway  length 
necessary  for  takeoff,  maintainability,  ferry  range,  and 
so  forth.  Possibilities  to  use  in  a  comparison  of  different 
types  of  tactical  aircraft  are  something  like  the  change  in 
the  outcome  of  the  projected  ground  battle  or  the  very 
popular  notion  of  the  forward  edge  of  the  battle  area  (the 
FEBA) .  'ltimately,  an  entire  profile  cf  measures  may  be 
requir* d  to  give  a  criterion  for  choice.  One  recent  pro¬ 
posal  for  a  study  of  the  mix  of  the  tactical  air  forces  in 
Europe  suggests  using  the  ''history’'  of  a  range  of  hypotheti¬ 
cal  waxs  in  which  NATO  is  the  defender,  comparing  the 
effectiveness  of  the  various  force  mixes  in  resisting  an 
overwhelming  attack  on  the  basis  of  time  indices — such  as 
the  time  it  takes  the  enemy  to  reach  the  Weser,  then  the 
Ems,  then  the  Rhine,  then  Paris;  the  time  it  takes  our 
side  to  reach  air  parity,  then  local  superiority,  then 
general  superiority,  then  air  supremacy.  There  are  fifteen 
indices  in  total,  the  others  involving  losses  and  force 
ratios. 

How  does  one  go  about  finding  a  satisfactory  measure 
of  effectiveness?  It  is  essentially  an  art;  a  matter  of 
trial  and  error  and  good  judgment.  It  may  be  hopelessly 
impossible  to  think  out  a  good  measure  in  advance.  The 
practical  way  may  be  to  take  a  rather  crude  value  scale, 
see  what  result  its  use  leads  to,  and  then,  if  that  result 


is  not  ir  accordance  with  common  sense,  revise  it.  For 
instance,  consider  the  slightly  apocryphal  example  of  the 
analyst  who  set  out  wo  use  a  linear  programming  model  to 
evolve  an  optimal  reducing  diet.  He  decided  that  an  ideal 
measure  of  effectiveness  would  be  to  get  the  most  volume 
within  the  constraints  of  the  necessary  nutrients  and 
calories.  Well,  he  put  his  model  on  the  machine,  ground 
away,  and  came  up  with  watermelon.  That  showed  him  he  had 
better  consider  dried  weight  to  exclude  water.  When  this 
was  done  he  came  up  with  bouillon  cubes,  which  consist 
largely  of  salt.  Since  this  would  not  be  very  palatable 
and  might  be  injurious  to  health,  he  made  still  another 
choice,  and  continuing  in  this  way,  he  finally  evolved  a 
satisfactory  rule  of  choice. 

THE  TREATMENT  OF  UNCERTAINTY 

Systems  analysis  is  concerned  with  problems  whose 
essence  is  uncertainty.  While  it  is  possible  to  forecast 
events  in  the  sense  of  mapping  out  possible  futures,  there 
is  no  way  to  predict  a  single  future  for  which  we  can  work 
out  the  best  system  or  determine  an  optimum  policy.  Con¬ 
sequently,  we  must  consider  a  range  of  possible  futures  or 
contingencies.  In  any  one  of  these  we  may  be  able  to 
designate  a  preferred  course  of  action,  but  we  have  no  way 
to  determine  such  an  action  for  the  entire  range  of  possi¬ 
bilities.  As  a  result,  defense  planning  is  rich  in  the 
kind  of  analysis  that  tells  what  damage  could  be  done  to 
the  United  States  given  a  particular  enemy  force  structure; 
but  it  is  poor  in  the  kinds  of  analyses  that  evaluate  how 
we  will  actually  stand  in  relation  to  our  possible  enemies 
in  years  to  come. 

It  is  not  enough  simply  to  acknowledge  that  uncertain¬ 
ties  exist  and  to  warn  that  some  things  have  been  left  out 
of  the  analysis  because  of  a  lack  of  Information,  for  such 
issues  may  have  a  critical  effect  on  the  conclusions.  The 


user  has  to  come  to  grips  with  these  issues  and  he  needs  to 
know  what  their  effects  must  be,  how  likely  they  are,  when 
he  can  expect  them,  and  what  he  can  do  about  them. 

Uncertainties  whose  probability  of  occurrence  is  more 
or  less  objective  or  calculable  can  be  handled  in  the  model 
by  Monte  Carlo  or  other  methods.  The  treatment  of  such 
uncertainty  is  a  considerable  practical  problem,  however, 
and  a  challenge  to  the  analyst.  The  pitfall  for  model- 
builders  lies  in  accepting  this  challenge — to  the  neglect 
of  the  real  or  unforeseeable  uncertainties.  These  typically 
involve  forms  of  ignorance  that  cannot  be  reduced  to  proba¬ 
bilities,  and  their  consequences  can  be  devastating.*  The 
objective  in  system  studies  is  not  to  learn  what  can  happen 
in  a  given  situation  with  a  specific  probability  as  the 
consequence  of  chance  fluctuation,  but  to  design  a  system 
or  determine  a  policy  so  that  any  fluctuations  are  unimpor¬ 
tant. 

What  can  *.?c  do  in  the  face  of  uncertainty?  One  pos¬ 
sibility  is  to  buy  time,  to  delay  decision  until  more  infor¬ 
mation  is  available.  A  second  is  to  buy  information,  to 
alleviate  our  uneasiness  by  data  collection.  Another  is  to 
buy  flexibility  as  a  hedge,  for  example,  by  developing  more 
than  one  alternative.  Sometimes  one  can  eliminate  alterna¬ 
tives  by  a  "dominance"  or  a  fortiori  argument.  Suppose  that 
in  a  particular  planning  decision  experienced  judgment 
strongly  favors  alternative  A,  but  the  analyst  feels  chat 
alternative  B  might  be  preferred.  In  performing  the  analy¬ 
sis,  one  may  choose  deliberately  to  resolve  the  major 
uncertainties  in  favor  of  A  and  then  see  how  B  compares 
under  these  adverse  conditions.  If  B  still  looks  good,  the 
case  in  favor  of  B  may  be  very  strong. 

In  addition,  sensitivity  analysis  and  contingency 
analysis  must  always  be  carried  out.  Let  me  illustrate  in 
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Particularly  those  uncertainties  that  depend  on  human 
caprice.  As  one  of  Damon  Runyan's  characters  (I  think)  put 
it:  "Nothing  what  depends  on  humans  is  worth  oads  of  irore 
than  8  to  3." 


terms  of  a  simple  hypothetical  and  unrealistic  example 
from  agriculture. 

A  farmer  must  decide  what  crop  or  crops  to 
plant  without  knowing  whether  the  weather  will 
be  wet,  moderate  or  dry.  An  analysis  is  per¬ 
formed  to  help  him  decide.  A  popular  approach 
is  employed  in  which  the  analysis  is  repeated 
in  turn  for  each  of  the  three  distinguishable 
types  of  weather,  in  each  case  determining  the 
best  crop  to  plant  for  that  type  of  weather. 
Considering  all  possible  crops,  it  is  found 
that  for  wet  weather  corn  would  be  best,  for 
moderate  weather  oats  would  be  best,  and  for 
dry  weather  wheat  would  be  best.  The  principal 
results  presented  to  the  farmer  consist  of  the 
findings  concerning  the  best  crops  in  the  three 
types  of  weather,  the  best  yields  achievable 
in  each  contingency  (i.e.,  the  wet-weather  yield 
of  corn,  the  moderate-weather  yield  of  oats,  and 
the  dry-weather  yield  of  wheat;,  and  estimates 
of  the  probabilities  of  wet,  moderate  and  dry 
weather.  The  implication  is  that  the  farmer 
ought  to  make  his  choice  from  the  "preferred" 
crops,  com,  oats  or  wheat,  or  perhaps  a  com¬ 
bination  of  these  to  provide  some  all-weather 
insurance. 

The  farmer  is  not  satisfied  with  the  analy¬ 
sis,  however.  He  points  out  that  the  analysis 
tells  him  what  crop  he  should  plant  if  he  knew 
for  certain  what  the  weather  would  be,  but  he 
doesn't  see  how  this  helps  him  to  decide  what 
to  plant  when  he  doesn't  know  what  the  weather 
will  be,  except  for  the  weather  probabilities. 

He  would  like  to  know,  for  example,  what  will 
happen  if  he  plants  com  and  the  weather  turns 
out  moderate  or  dry,  and  similarly  for  the  other 
crops.  The  analyst  therefore  prepares  a  two- 
way  contingency  table,  showing  for  each  of  the 
three  "preferred"  crops  the  yields  in  wet, 
moderate  and  dry  weather.  Yields  for  various 
mixtures  of  these  crops  are  also  shown  in  the 
various  tyoes  of  weather.  It  is  found  that 
each  of  the  three  crops  is  rather  narrowly 
tailored  for  that  type  of  weather  in  which  it 
is  best,  and  elves  disastrously  poor  yields  in 
other  types  of  weather.  Oats,  for  example, 
gives  very  poor  yields  in  wet  or  dry  weather, 
but  very  good  yields  in  moderate  weather.  The 
farmer  can  insure  against  disaster  by  planting 
a  mixture  of  corn,  oats  and  wheat,  thereby 
obtaining  a  fair  overall  yield  whatever  the 
weather . 
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The  farmer  is  still  not  satisfied,  however. 

The  contingency  table  does  give  him  the  information 
he  wants  on  the  three  ’’preferred"  crops,  but  he  would 
like  to  see  the  same  information  for  some  other  crops, 
even  though  they  have  been  ruled  out  as  "inferior" 
in  the  analytical  optimization.  The  analyst 
obligingly  expands  the  contingency  tabic  to  show 
the  yields  of  various  other  "inferior"  crops  in 
wet,  moderate  and  dry  weather.  At  this  point 
it  is  noted  that  cane,  which  is  inferior  to  corn 
in  wet  weather,  inferior  to  oats  in  moderate 
weather,  and  inferior  to  wheat  in  dry  weather, 
gives  a  "pretty  good"  yield  in  all  types  of 
weather,  providing  better  all-weather  insurance 
than  can  be  achieved  with  any  combination  of 
the  three  "preferred"  crops.  This  particular 
farmer,  having  a  pronounced  aversion  to  risk, 
decides  that  of  all  the  crops  he  prefers  the 
weather— yield  pattern  of  cane  over  that  of  any 
other  crop  or  combination  of  crops.  Another 
farmer,  looking  at  the  same  table,  might  prefer 
to  take  somewhat  more  of  a  chance  on  alfalfa, 
another  "inferior"  crop  shown  to  give  a  rather 
good  yield  in  wet  or  moderate  weather  but  a 
poor  yield  in  dry  weather.  Still  another  might 
prefer  to  take  a  greater  chance  on  com,  but 
not  necessarily  because  it  was  one  of  the  "pre¬ 
ferred"  crops  in  the  original  analysis. 

The  first  approach  in  the  example  above, 
which  narrows  down  the  alternative  systems  to 
those  which  are  "preferred"  for  each  of  the 
various  situations  or  sets  of  assumptions  about 
the  uncertain  factors,  is  far  from  uncommon 
in  actual  applications  of  systems  analysis. 

It  can  be  useful  in  indicating  some  of  the 
systems  which  merit  consideration  by  the  planner 
or  decisionmaker.  It  can  be  worse  than  useless, 
however,  if  it  leads  the  customer  to  limit  his 
attention  only  to  these  "preferred"  systems. 

In  the  example  aoove,  the  farmer  was  not  so 
misled;  in  practice,  however,  there  is  a  real 
danger  that  the  customer  will  be  misled  into 
confining  his  choice  to  an  unduly  narrow  menu 
of  alternatives. 

The  approach  evolved  toward  the  end  of  the 
example  above  has  the  advantage  of  not  ruling 
out  system s  which  the  customer  ought  to  consider. 

It  has  the  disadvantage,  however,  of  not  ruling 
out  very  many  systems  at  all;  it  eliminates 
only  those  "inefficient"  systems  which  are  in¬ 
ferior  or  at  most  equal  to  other  systems  in  all 
situations  or  for  all  assumptions  about  uncer¬ 
tain  factors.  In  many  cases  the  remaining 
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"efficient"  systems  are  too  numerous  to  present 
to  the  customer;  some  means  of  narrowing  them 
down  further  to  a  manageably  small  list  must 
be  found  if  the  analysis  is  to  be  useful  to  the 
customer.  This  may  require  going  beyond  the 
bounds  of  strictly  legitimate  analysis,  by  such 
expedients  as  optimizing  on  the  basis  of  weighted 
values  of  performance  in  the  various  situations 
or  for  the  various  assumptions  about  the  uncer¬ 
tain  factors,  or  eliminating  systems  by  the  direct 
application  of  the  analyst's  judgment.  The  an¬ 
alyst  cannot  avoid  employing  judgment  in  initially 
formulating  the  analysis  and  in  initially  limiting 
consideration  to  some  subset  of  systems  out  of 
the  infinity  of  all  conceivable  systems;  further 
employments  of  judgment  later  in  the  analysis 
may  also  be  unavoidable. 

Whatever  approach  is  used  in  narrowing 
down  the  list  of  systems  to  be  presented  to  the 
customer,  the  approach  should  be  described  as 
explicitly  as  possible.  The  presentation  should 
include,  among  other  things,  a  contingency  table, 
showing  for  each  system  its  performance  and  cost 
in  each  of  the  various  relevant  situations  and/or 
for  each  set  of  assumptions  about  the  uncertain 
factors.  Digesting  this  information  and  using 
it  in  making  decisions  or  plans  puts  a  heavy 
burden  on  the  decisionmaker  or  planner,  but  it 
can't  be  helped.  Systems  analysis  does  not 
relieve  the  customer  of  the  responsibility  for 
facing  the  uncertain  consequences  of  his  de¬ 
cisions  or  plans;  it  can,  however,  help  hin 
face  uncertainty  with  a  better  appreciation  of 
the  relevant  considerations  than  he  might 
otherwise  have  had." 


HANDLING  THE  NONQUANTIFIABLES 

One  obvious  limit  ;.tion  is  that  for  broad  questions  the 
analysis  is  necessaril;  incomplete.  Time  and  money  place 
sharp  limits  on  how  fa;  any  inquiry  can  be  carried.  Almost 
all  systems  analyses  must  stop  far,  far  short  of  completion 
either  for  lack  of  funds,  of  time,  or  of  justification  for 
spending  further  funds  or  time  on  them. 

Still  more  important,  however,  is  the  fact  that  even 
with  no  limitations  of  time  and  money  analysis  can  never 
treat  all  the  considerations  that  may  be  relevant.  Some  are 
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too  intangible — for  example,  how  some  unilateral  U.S. 
action  will  affect  MATO  solidarity,  or  whethc  Congress 
will  accept  e  nomies  that  disrupt  cherished  institutions 
such  as  the  National  Guard  or  radically  change  the  pattern 
of  domestic  spending.  Considerations  of  this  type  c*n 
play  as  important  a  role  in  the  choice  of,  for  example, 
alternative  military  policies  as  any  idealized  cost- 
effectiveness  calculations.  But  ways  to  measure  these 
considerations  even  approximately  do  not  exist  today,  and 
chey  must  be  handled  intuitively.  Other  considerations 
involve  moral  judgments — for  example,  whether  national 
security  is  better  setved  by  an  increase  in  the  budget 
for  defense  or  for  welfare,  or  under  what  circumstances 
the  presewation  of  an  immediate  advantage  is  worth  the 
compromise  of  fundamental  principles.  The  analyst  can, 
at  the  very  least,  /’.pply  his  own  judgment  and  intuition 
to  these  considerations,  thus  insuring  that  they  are  brought 
to  the  attention  of  the  man  with  the  responsibility  for  the 
decision.  The  problem  is  to  do  something  more. 

When  onsiderations  of  this  type  dominate  and  we  can¬ 
not  adequately  approximate  the  situation  on  a  computer  or  by 
mathematical  equations,  a  different  approach  is  required, 
one  that  directly  involves  experts  whose  knowledge  should  be 
of  help  In  attacking  intangible  and  moral  considerations. 

Ordinarily,  we  would  like  to  have  the  advice  of  several 
experts.  The  traditional  way  to  try  for  their  consensus 
has  been  to  assemble  them  as  a  committee,  to  let  them  dis¬ 
cuss  the  problem  freely,  and  to  require  that  they  arrive  at 
a  joint  answer.  Committees,  however,  often  fAll  to  make 
their  assumptions  and  reasoning  explicit,  siuce  their  find¬ 
ings  may  be  obtained  through  bargaining.  Better  judgments 
can  be  obtained  by  asking  the  exnerts  to  work  in  a  group 
exercise,  which  might  range  from  a  simple  structured  dis¬ 
cussion  to  an  "operational  game." 

This  latter  approach  is  a  fore,  of  simulation  involving 
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role  playing  by  human  subjects.  Here  the  game  structure 
— that-  is  the  nodel — furnishes  the  participants  with  an 
artificial,  simulated  environment  within  which  they  can 
jointly  and  simultaneously  experiment,  acquiring  through 
feedback  the  insights  necessary  to  nak.i  successful  pre¬ 
dictions  within  the  gaining  context  and  thus  indirectly 
about  the  real  world.  For  example,  to  study  the  economy 
of  an  underdeveloped  country,  the  various  sections  of  the 
economy  might  be  simulated  by  specialized  experts.  These 
experts  o**  specialists  would  be  expected,  in  acting  out 
their  roles,  to  use  their  intuition  as  experts  to  simulate 
as  best  they  could  the  attitudes  and  consequent  decisions 
of  their  real-life  counterparts.  For  instance,  a  player 
simulating  a  goods— producing  sector  of  the  economy  might, 
within  constraints,  shut  down  or  expand  manufacturing 
facilities,  modernize,  change  raw  material  and  labor  inputs, 
vary  prices,  and  so  on.  There  would  also  need  to  be  govern¬ 
ment  players,  who  could  introduce  new  fiscal  or  monetary 
policies  and  regulations  (affecting  taxes,  subsidies,  tar¬ 
iffs,  price  ceilings,  etc.),  as  well  as  social  and  politi¬ 
cal  innovations  with  only  indirect  economic  implications 
(social  security,  education,  appeals  to  patriotism,  uni¬ 
versal  military  service,  etc.).  In  addition,  there  would 
hove  to  be  a  control  team  to  lay  down  the  rules  governing 
the  players'  options  and  constraints  and  to  evaluate  the 
actions  taken  within  these  rules. 

An  important  need  in  all  analysis  is  to  set  the  con¬ 
text.  One  approach  is  what  military  systems  analysts  call 
"scenario  writing."  This  is  an  effort  to  show  how  a  future 
state  of  affairs  might  evolve  out  of  a  present  one.  A 
scenario  is  thus  a  primitive  model  that  helps  define  the 
relevant  environments  or  situations  in  which  to  test  the 
alternatives.  A  collection  of  scenarios  provides  an  in¬ 
sight  into  how  future  trends  can  depend  on  factors  under 
our  control  and  suggests  policy  options  to  us. 
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Another  type  of  group  action  that  can  play  the  role 
of  the  model  when  we  can*t  build  a  satisfactory  one,  some¬ 
what  less  structured  than  the  operational  game,  attempts 
to  improve  the  panel  or  committee  approach  by  subjecting 
the  views  of  individual  experts  to  each  other's  criticism 
without  actual  confrontation  and  its  possible  psychological 
shortcomings.  In  this  approach,  called  the  Delphi  method 
by  its  principal  contributor  Olaf  Helmer,  direct  debate  is 
replaced  by  the  interchange  of  information  and  opinion 
through  a  carefully  designed  sequence  of  questionnaires. 

At  each  successive  interrogation,  the  participants  are 
given  new  and  refined  information  in  the  form  of  opinion 
feedback,  which  is  derived  by  a  computed  consensus  fron 
the  earlier  pari  of  the  program.  The  process  continues 

until  either  a  cons  '.sus  is  determined,  or  the  conflicting 

* 

views  are  documented  fully. 

In  addition  to  the  problems  inherent  in  ^11  policy 
analysis,  there  are  additional  problems  we  create  for  our¬ 
selves.  Among  others,  these  have  to  do  with  introducing 
bias;  attacking  the  wrong  problem;  and  forgetting  the  pro¬ 
per  role  of  the  analysis. 

BIAS 

The  most  serious  error  in  analysis  is  to  look  at  an  un¬ 
duly  restricted  range  of  alternatives.  Although  narrowing 
the  range  of  choice  certainly  makes  the  evaluation  easier,  we 
may  pay  a  high  price  if  some  of  the  excluded  alternatives  are 
better  than  those  remaining. 

The  most  frequent  cause  of  failure  to  look  at  the  full 
range  of  alternatives  is  an  "attention  bias."  This  often 
takes  the  form  of  an  unconscious  adherence  to  a  "party  line" 
or  "cherished  belief."  All  organizations  foster  one  to  some 
extent;  the  military  services,  the  civilian  agencies  of  the 
government,  and  private  businesses  are  no  exception.  Ex¬ 
perience  suggests  that  Herman  Kahn  was  right  when  he  called 
the  party  line  "the  most  important  single  reason  for  the 


For  additional  discussion,  see  0.  Helmer,  Social 
Technology,  The  RAND  Corporation,  P-3063,  February  1965. 
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tremendous  miscalculations  that  are  made  in  foreseeing  and 

preparing  for  technical  advances  or  changes  in  the  strategic 
* 

situation.  Examples  are  plentiful:  the  general  whose 
gaze  is  so  fixed  on  "winning"  a  local  war  militarily  that 
he  excludes  other  considerations;  or  the  statesman  so  con¬ 
vinced  that  peace  can  be  maintained  through  deterrence  that 
he  completely  disregards  what  might  happen  should  deterrence 
fail;  or  the  weaponeer  who  is  so  fascinated  by  startling 
new  weapons  that  he  assumes  they  can  of  course  be  used. 

The  history  of  strategic  bombing  studies  since  World 
War  II  illustraces  the  workings  of  this  influence.  In 
World  War  II ,  the  bombing  problem  was  to  penetrate  the 
defenses,  bomb  accurately,  and  return.  The  bomber's  con¬ 
cern  was  with  enemy  fighters,  antiaircraft  guns,  and 
missiles — not  with  enemy  bombers.  For  years,  even  long 
after  the  Soviets  were  expected  to  have  nuclear  weapons, 
studies  paid  no  serious  attention  to  the  possibility  that 
our  bombers  might  be  vulnerable  on  the  ground,  and  did  not 
consider  an  attack  on  the  enemy's  grounded  bombers  as  a 
strategy.  Requirements  and  specifications  for  future  bombers 
hardly  conside '3d  the  problem  of  surviving  the  enemy  offense. 
This  was  not  stupidity  on  the  part  of  anyone.  For  instance, 
in  the  Navy-6AC  controversy  over  the  B— 36  in  1949,  the  Navy 
challenged  the  B^-36  on  every  basis  it  could  think  of — 
including  the  argument  that  strategic  bombing  was  immoral 
— but  the  question  of  its  vulnerability  on  the  ground  did 
not  come  up.  RAND  strategic  bombing  studies,  even  after 
the  seriousness  of  the  problem  of  bomber  survival  on  the 
ground  had  been  pointed  out,  continued  to  concentrate  on 
such  questions  as  speed,  altitude,  low  versus  high  penetra¬ 
tion,  supersonic  dash,  bombing  altitude,  small  versus  large 
planes,  and  what  targets  to  select.  Thus,  in  a  1952  com¬ 
parison  of  aircraft  with  missiles  for  strategic  bombing 
we  ignored  any  effect  on  the  comparison  of  possible  dif¬ 
ferences  in  ground  survivability — and  no  one  we  briefed 

*H.  Kahn  and  I.  Mann,  Ten  Common  Pitfalls,  The  RAND 
Corporation,  RM-1937,  July  1957,  p.  42. 
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took  us  to  task  for  this  omission'.  The  Russians  and  the 
British  took  even  longer  than  we  did  to  recognize  ground 
vulnerability,  and  some  people  think  the  French  have  not 
yet  really  absorbed  the  idea.  It  took  the  extensive  brief¬ 
ings  of  a  RAND  study  devoted  to  proving  that  base  vulner¬ 
ability  was  a  serious  threat  to  national  security  to  get 
major  attention  on  the  problem. 

A  party  line  can  be  most  influential  in  shaping  the 
study  during  the  early  stages.  What  can  happen  is  that  the 
participants  and  successive  reviewers  become  aware  that 
some  of  the  alternatives  or  certain  assumptions  being  con¬ 
sidered  are  frowned  on  by  higher  ranking  officers.  It 
seems  useless,  even  hazardous,  to  support  such  unpopular 
views  strongly  and  gradually  they  may  be  stressed  less  and 
less  or  even  forgotten.  In  its  most  extreme  form,  this 
influence  can,  in  effect,  cause  the  analyst  to  lose  his 
independence  of  view. 

A  similar  pitfall  is  to  expect  a  man  or  the  organiza¬ 
tion  that  created  a  system  to  discover  its  faults.  Either 
may  fail  completely  to  take  into  account  some  technical 
notion  or  fact.  An  analyst  is  sometimes  shocked  to  discover 
that  other  organizations  regard  the  system  he  is  investi¬ 
gating  as  controversial  at  best,  or  that  they  even  consider 
his  own  people  to  be  completely  wrong.  For  example,  he  may 
find  out  that  simple  countermeasures  can  be  devised  to 
render  the  system  almost  worthless.  As  an  example,  consider 
the  English  invention  of  "window”  or,  as  we  call  it  here, 
"chaff."  In  1937,  in  Great  Britain,  the  merits  of  radar 
and  infrared  were  being  argued.  Talk  about  shutting  down 
research  on  ’.nfrared  because  of  its  vulnerability  to  counter¬ 
measures  started.  It  was  then  that  R.  V.  Jones,  a  physicist 
who  was  working  on  infrared,  came  up  with  the  idea  of 
"window"  to  show  (in  his  words)  "They  weren't  so  invulner¬ 
able  themselves."  Wataon-Watt,  the  English  radar  pioneer, 
apparently  never  had  quite  the  heart  to  urge  trials  for 
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Jones'  device,  which  seemed  so  certain  to  spoil  his 
"beautiful”  invention.  Sometime  later,  others  had  the 
trials  carried  out. 

The  first  step,  and  the  most  effective  one,  in  crim¬ 
inating  bias  is  to  recognize  that  it  is  likely  to  exist. 

The  second  is  to  have  the  analysis  critiqued  by  people 
with  a  different  set  of  cherished  beliefs  or  party  line. 

To  appraise  a  system  and  discover  such  tilings,  good  analysts 
obligate  themselves  to  consult  people  w*  th  an  adverse  opin¬ 
ion  of  the  worth  of  the  system,  largely  because  they  know 
how  hard  it  is  to  get  a  scientist  or  engineer  to  display 
much  ingenuity  in  tearing  down  a  technically  brilliant 
design  that  he  has  been  working  on  for  years. 

Another  manifestation  of  parochialism  in  analysis 
occurs  when  service  A  proposes  a  future  weapon  system  with 
fabulous  properties  for  victory  and  compares  it  with  the 
semiobsolete  current  equipment  of  service  B  rather  than 
with  the  new  system  that  B  hopes  to  introduce,  or  uses  a 
"scenario"  ideally  suited  for  making  the  good  points  of  A 
apparent . 

PROBLEM  FORMULATION 

It  is  all— important  to  tackle  the  "right"  problem. 

A  large  part  of  the  investigators'  effort  must  be  invested 
in  thinking  about  the  problem,  exploring  its  proper  breadth, 
and  trying  to  discover  the  appropriate  objectives  and  to 
search  out  good  criteria  for  choice.  If  we  have  not  chosen 
the  right  set  of  alternatives  to  compare,  we  will  not  dis¬ 
cover  the  best.  But  if  we  have  chosen  the  wrong  objective, 
then  we  might  find  a  solution  to  the  wrong  problem.  Get¬ 
ting  an  accurate  answer  to  the  wrong  question  is  likely  to 
be  far  less  helpful  than  an  incomplete  answer  to  the  right 
question. 

A  serious  pitfall  is  the  failure  to  allocate  the  total 
time  deliberately  so  that  a  sufficient  share  of  it  will  be 
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spent  in  deciding  chat  the  problem  really  is.  It  is  a 
ptifall  to  give  in  to  the  tendency  to  "get  started"  with¬ 
out  having  devoted  a  lot  of  thought  to  the  problem. 

In  the  firsf  systems  analysis  I  worked  on  I  fell  into 
this  pitfall.  The  analysis  was  being  carried  out  to  help 
in  the  design  of  a  strategic  bomber  for  the  mid  1950s. 

One  moot  question  was  whether  or  not  to  put  a  tail  turret 
with  machine  guns  on  the  bomber.  The  day  I  reported  tor 
work  it  was  suggested  that  I  try  to  determine  how  the  proba¬ 
bility  of  hitting  an  attacking  fighter  was  affected  by  the 
correlation  between  aiming  points  in  air-to-air  machine  gm 
fire — something  always  present  because  the  shots  are  not 
aimed  independently.  I  saw  an  interesting  mathematical 
problem,  and  started  right  to  work.  Several  months  later, 
after  we  had  found  a  satisfactory  approximation  and  checked 
it  by  a  Monte  Carlo  simulation,  I  saw  some  test  firings  of 
an  early  version  of  an  air-to-air  guided  rocket.  I  was 
much  impressed.  Not  until  some  time  later,  however,  did 
it  suddenly  dawn  on  me  that  by  the  time  the  bombers  we  hau 
under  study  could  become  operational,  they  would  not  have 
an  opportunity  to  fire  back  at  fighters  with  machine  guns. 
Thinking  about  tue  problem  as  a  whole  and  asking  a  few 
questions  would  very  likely  have  saved  all  of  us  a  lot  of 
time . 

Systems  studies  have  resulted  in  rather  important 
changes,  not  only  in  how  the  policymaker  carries  out  his 
activity,  but  in  the  objectives  themselves;  hence  it  would 
be  self-defeating  to  accept  without  inquiry  the  sponsors' 
view  of  what  the  problem  is.  But  how  is  the  analyst  to 
know  that  his  formulation  of  the  problem  is  superior? 

H*s  only  possible  advantage  lies  in  analysis.  That  is, 
the  process  of  problem  formulation  itself  has  to  be  the 
subject  of  analysis.  What  this  means  is  that,  using  the 
few  facts  and  relationships  that  are  known  at  this  early 
stage  and  assuming  others,  the  analyst  must  simply  make  an 


-17- 


attempt  to  solve  the  problem.  It  is  this  attempt  that 
will  give  turn  a  basis  for  better  formulation. 

Let  me  offer  an  example.  For  fiscal  year  1952,  Con¬ 
gress  authorized  approximately  $3  .5  bilJion  for  air  base 
construction,  about  half  to  be  spent  overseas.  RAND  was 
asked  to  suggest  ways  to  acquire,  construct,  and  maintain 
air  bases  in  foreign  countries  at  minimum  cost.  The 
analyst  who  took  on  this  problem  accepted  it  as  given  as 
essentially  one  of  logistics.  He  spent  a  long  time — several 
months,  in  fact — thinking  about  it  before  he  organized  a 
study  team.  Although  he  had  little  of  the  information 
needed  to  make  recommendations,  he  began  to  look  at  the 
problem  in  relation  to  the  Air  Force  as  a  ’whole.  He 
argued  that  base  choice  would  critically  affect  the  com¬ 
position,  destructive  power,  and  cost  of  the  entire  stra¬ 
tegic  force  and  thus  that  it  was  not  wise  to  rest  a  decision 
on  base  structure  and  location  merely  on  economy  in  base 
cost  alone.  He  came  to  the  conclusion  that  the  real  pro¬ 
blem  was  not  one  of  the  logistics  of  foreign  air  bases, 
but  the  much  broader  one  of  where  and  how  to  base  the 
Strategic  Air  Force  and  how  to  operate  it  in  conjunction 
with  the  base  system  chosen.  His  views  prevailed,  and  he 
led  the  broader  study.*  An  Air  Force  Committee  later 
estimated  that  the  study  recommendations  saved  over  $1 
billion  just  in  construction  costs.  In  addition,  it  sparked 
a  tremendous  improvement  in  strategic  capability,  particu¬ 
larly  with  regard  to  survival. 

In  analysis,  the  problem  never  remains  static.  Inter¬ 
play  between  a  growing  understanding  of  what  it  involves 
now  and  might  involve  in  the  future  forces  a  constant 
redefinition.  Thus,  the  study  just  mentioned,  originally 
conceived  as  an  exercise  to  reduce  costs,  became  in  the 
end  a  study  of  U.S.  strategic  deterrent  policy.  Its  first 

For  the  full  report,  see  A.  J.  Wohlstetter,  F.  S. 
Hoffman,  R.  J.  Lutz,  and  H.  S.  Rowen,  Selection  and  Use  of 
Strategic  Air  Bases,  The  RAND  Corporation,  R— 2bb,  April  1954. 
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recommendation  was  to  reduce  SAC  '  jlnerabili ty  by  cutting 
sharply  the  number  of  functions  \  rformed  at  overseas  bases 
and  thus  the  time  during  which  tl  force  was  exposed  on 
these  bases.  That  this  also  reduced  costs  was  secondary. 

THE  ROLE  OF  ANALYSIS 

The  role  of  systems  analysis  is  to  sharpen  the  intui¬ 
tion  and  judgment  of  the  decisionmakers;  to  provide  advice 
concerning  a  particular  question.  Problems  arise  if  we 
forget  these  things. 

It  is  a  dangerous  pitfall  to  attempt  to  set  up  a  model 
that  treats  every  aspect  of  a  complex  problem  simultaneously. 
What  can  happen  is  that  the  analyst  finds  himself  criticized 
because  the  first  model  he  has  selected  has  left  out  various 
facets  of  the  situation  being  investigated.  He  is  vulnerable 
to  these  criticisms  if  he  doesn't  realize  that  bhe  question 
being  asked,  as  well  as  the  process  being  represented  must 
determine  the  model.  Without  attention  to  the  question,  he 
has  no  rule  for  guidance  as  to  what  to  accept  or  reject;  he 
has  no  real  goals  in  view  and  no  way  to  decide  what  is  im¬ 
portant  and  relevant.  He  can  answer  criticism  only  by  making 
the  model  bigger  and  more  complicated.  This  may  not  stop 
the  criticisms,  for  something  must  always  be  left  out.  The 
size  of  the  model  is  then  determined  not  by  what  is  really 
relevant  but  by  the  capacity  of  the  computing  machine. 

Of  course,  it  is  easy  for  the  analyst  to  become  more 
interested  in  the  model  than  in  the  problem  itself.  Tech¬ 
nical  people  with  specific  training,  knowledge,  and  capabil¬ 
ity  like  to  use  their  talents  to  the  utmost.  It  is  easy 
for  analysts  to  focus  attention  on  the  mechanics  of  the 
computation  or  on  the  technical  relationships  in  the  model 
rather  than  on  the  important  questions  raised  in  the  study. 
They  may  tvus  find  out  a  great  deal  about  the  inferences 
that  can  be  drawn  from  the  model,  but  very  little  about 
the  question  they  set  out  to  answer.  But  the  role  of  the 
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analyst  is  to  provide  advice,  not  to  build  new  tools  or 
to  advance  the  art  of  operations  research. 

A  serious  pitfall  is  to  believe  that,  if  the  work  is 
done  correctly,  systems  analysis  is  unlimited  in  the  advice 
it  can  supply  a  decisionmaker.  This  is  far  from  the  case. 

No  model  can  substitute  for  the  decisionmaker — the  limita¬ 
tions  on  analysis  mentioned  earlier  prevent  this.  However, 
it  has  been  proposed  a  number  of  times  (ever,  to  the  extent 
of  writing  a  study  contract)  that  a  general  computer  model 
for  strategic  war  be  set  up  to  supply  weapon  designers  with 
a  systematic  evaluation  Oj.  their  design  concepts  and  to 
enable  the  Department  of  Defense  to  evaluate  the  worth  of 
alternative  "design  solutions"  developed  by  competing 
contrac tors . 

One  argument  for  such  a  model  notes  that  "the  choice 
of  assumptions,  the  forecast  of  the  future,  and  the  methods 
of  analysis  have  a  marked  influence  on  the  performance  and 
physical  characteristics  of  the  weapon  system  set  forth  as 
preferred  or  optimal";  therefore,  a  uniform  framework 
would  mean  that  "the  results  obtained  by  the  various  con¬ 
tractors  would  be  comparable  since  the  effect  due  to 
variation  in  the  assumptions  they  might  have  chosen  to 
form  their  models  would  have  been  eliminated."  This  may 
indeed  be  the  case,  but  will  the  end  result  be  desirable? 

A  rigidly  specified  framework  may  mitigate  one  sort  of 
undesirable  bias — by  making  it  difficult  for  an  analysis 
to  be  used  to  rationalize  conclusions  already  otherwise 
derived — but  only  at  the  severe  risk  of  introducing  other 
biases. 

A  fundamental  objection  is  that  a  u.tiform  framework 
necessarily  conceals  or  removes  by  assumption  many  extremely 
important  uncertainties,  and  therefore  tends  to  lead  to 
solutions  that  disregard  the  value  of  hedging  against  those 
uncertainties.  Another  is  that  even  if  efforts  were  made 
to  Keep  the  model  up  to  date,  this  would  turn  out  to  be 
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impossible,  for  the  analyst  must  be  able  to  modify  his 
model  in  the  terminal  stages  of  his  study  to  accommodate 
information  acquired  during  the  early  phases.  Indeed,  in 
a  problem  involving  the  struggle  between  nations,  there 
are  so  many  factors  of  shifting  importance,  and  such 
radical  changes  in  objectives  and  tactics  are  likely, 
that  most  models  are  obsolete  long  before  the  recommenda¬ 
tions  from  the  study  can  become  accepted  policy.  Moreover, 
if  a  model  or  a  mathematical  formula  were  used  to  indicate 
which  proposal  to  select,  the  proposers'  emphasis  would 
soon  focus  on  how  to  make  his  design  look  good  in  terms  of 
this  analytic  definition,  not  on  how  to  make  it  look  good 
against  the  enemy— -a  much  harder  problem. 

CONCLUDE  REMARKS 

How  can  systems  analysis  provide  better  advice  than 
other  more  traditional  ways  to  furnish  advice — say,  by  an 
expert  or  a  committee?  In  areas  such  as  national  defense 
planning,  where  there  is  no  accepted  theoretical  foundation, 
advice  obtained  from  experts  working  individually  or  as  a 
committee  must  of  course  depend  largely  on  judgment  and 
intuition.  So  must  the  advice  from  systems  analysis.  But 
the  virtue  of  systems  analysis  is  that  it  permits  the  judg¬ 
ment  and  intuition  of  the  expercs  in  relevant  fields  to  be 
combined  systematically  and  efficiently.  The  essence  of 
the  method  is  to  construct  and  operate  within  a  "model,*' 
a  simplified,  stylized  representation  of  the  real  situation 
appropriate  to  the  question.  Such  a  model,  which  may  take 
such  varied  forms  as  a  set  of  mathematical  equations,  a 
computer  simulation,  an  operational  game,  or  even  a  purely 
verbal  description  of  the  situation  in  the  form  of  a 
"scenario,"  introduces  a  precise  structure  and  terminology 
that  serve  primarily  as  an  effective  means  of  communica¬ 
tion,  enabling  the  participants  in  the  study  to  exercise 
their  judgment  and  intuition  in  a  well-defined  context  and 
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in  proper  relation  to  the  judgment  and  intuition  of  others. 
Moreover,  through  feedback  from  the  model  (the  results  of 
computation,  the  countermoves  in  the  game,  or  the  critique 
of  the  scenario),  the  experts  have  a  chance  to  revise  early 
judgments  and  thus  arrive  at  a  clearer  understanding  of  the 
problem  and  its  context,  and  perhaps  of  their  subject  matter. 

As  Charles  Hitch  put  it:  "Systems  analysis  should  be 
looked  upon  .  .  .  as  a  framework  which  permits  the  judgment 
of  experts  in  numerous  subfields  to  be  utilized — to  yield 
results  which  transcend  any  individual  judgment.  This  is 
its  aim  and  opportunity." 

The  process  has  problems  and  limitations,  but  the 
alternatives  to  analysis  have  the  same  and  even  greater 
ones . 
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